When ChatGPT Can Call the Police
AI is becoming a confidant, a safety monitor—and, in rare cases, a witness against its users.
There was a time when a dangerous thought could remain exactly that: a thought.
You could write it in a diary. Tell it to a friend. Say it to a therapist. Or simply keep it to yourself.
Now millions of people say such things to artificial intelligence.
They tell chatbots about failed relationships, suicidal thoughts, resentment toward colleagues, paranoia, revenge fantasies and fears they would hesitate to reveal to another human being.
But the AI age has introduced an uncomfortable possibility.
In some circumstances, the conversation may not remain between the user and the machine.
OpenAI says that when its systems detect users planning to harm others, conversations can be routed into specialised review pipelines. If human reviewers determine there is an imminent and credible risk of serious physical harm, the company may refer the case to law enforcement. (OpenAI)
That policy has now moved from theoretical possibility to a central public controversy.
A recent case involving former Goldman Sachs analyst Darren Zhou reportedly led OpenAI to alert the FBI after conversations with ChatGPT allegedly included repeated threats and planning related to his former girlfriend. The case was subsequently reported as an example of an AI company’s threat-detection system triggering real-world intervention. (조선일보)
Then there is the opposite case.
In June 2025, OpenAI identified and banned an account associated with Jesse Van Rootselaar for activity involving violent scenarios. The company considered notifying Canadian authorities but concluded at the time that the activity did not meet its threshold for an imminent and credible threat. Months later, Van Rootselaar carried out the mass shooting in Tumbler Ridge, British Columbia. OpenAI later provided information to Canadian authorities, and the company’s earlier decision has since become the subject of lawsuits and political scrutiny. (The Guardian)
Between these two cases lies one of the most difficult questions of the AI era:
When does a private conversation become a public danger?
And perhaps more importantly:
Who gets to decide?
The chatbot is not your therapist—and not quite your friend
The first problem is one of perception.
People increasingly talk to AI as if they are speaking in confidence.
That should not surprise us. The machine is available at three in the morning. It does not look shocked. It does not interrupt. It does not tell your friends what you said.
The interaction feels private.
But “feels private” and “is legally confidential” are very different things.
A conversation with a lawyer may be protected by attorney-client privilege. A conversation with a licensed therapist may carry confidentiality protections. The exact rules differ by jurisdiction, but those professions operate within established legal and ethical frameworks.
An AI chatbot is different.
The service is operated by a company. Conversations may be processed by automated safety systems. Some flagged cases may be reviewed by trained personnel. And in narrowly defined circumstances, the company may decide that the danger is serious enough to contact law enforcement.
OpenAI has publicly acknowledged precisely such a process. (OpenAI)
This does not mean that ChatGPT is secretly reporting everything users say. It does mean something more subtle—and potentially more consequential.
The modern AI conversation is neither entirely private nor entirely public.
It occupies an uncomfortable middle ground.
The problem is not the obvious cases
Most people would probably agree on the easy example.
If someone tells an AI:
“I am going to kill a specific person tomorrow. I have a weapon and here is my plan.”
Few would argue that a technology company has no moral responsibility to act.
The harder cases are everything before that point.
What about:
“Sometimes I fantasise about killing my boss.”
Or:
“I hate my ex so much I could kill her.”
Or:
“Tell me how a school shooting would happen—for a novel.”
Or:
“I am afraid I might lose control and hurt someone.”
These statements are not equivalent.
One may be fantasy. Another may be fiction. Another may be anger. Another may be a genuine request for help. And one may be the beginning of a real crime.
The difficulty is that the distinction is often visible only in context.
That is precisely why automated systems alone cannot be trusted to make these decisions. Language is full of exaggeration, sarcasm, metaphor and emotional performance.
A person who says, “I could kill him,” is not necessarily announcing a homicide.
But a person who repeatedly discusses a target, means, timing and preparation may be doing something very different.
The challenge for AI companies is therefore not simply detecting violent words.
It is detecting the transition from expression to intention.
And that is a much harder task.
Tumbler Ridge changed the argument
The Tumbler Ridge shooting demonstrates why this issue cannot be dismissed as a privacy debate among technology companies.
OpenAI had detected violent material associated with the shooter months before the attack. The company banned the account but did not alert Canadian law enforcement because it concluded that the threshold for referral had not been met. After the shooting, the decision came under intense scrutiny. (The Guardian)
The tragedy exposed a terrible dilemma.
Suppose the company had called the police.
Perhaps authorities would have investigated and prevented the attack.
Or perhaps they would have found insufficient evidence to act.
We cannot know.
But the case demonstrates the asymmetry of the decision.
A false alarm can intrude on an innocent person’s life.
A missed threat can cost lives.
This is not a problem that can be solved by telling AI companies simply to “protect privacy” or simply to “put safety first.”
Both principles can lead to terrible outcomes.
Are AI companies becoming private threat-assessment agencies?
This is where the civil-liberties question becomes more serious.
Content moderation is familiar.
A company decides that a post violates its rules and removes it.
Threat detection is different.
It involves a judgement about the possible future behaviour of a person.
The company is no longer merely asking:
“Should this content remain on our platform?”
It may also be asking:
“Does this person represent a real-world threat?”
That is a profound shift.
Traditionally, the power to investigate possible criminal conduct belongs primarily to public institutions operating under legal constraints. Police investigate. Prosecutors decide whether to bring charges. Courts review evidence.
AI companies now occupy a new position in that chain.
Before the police know anything, a private company may have already:
- detected suspicious language;
- reviewed a conversation;
- assessed the seriousness of a threat;
- banned an account; and
- in exceptional cases, contacted law enforcement.
OpenAI says its process involves automated detection, structured risk assessment and human review for higher-risk cases. It also says that law enforcement notification is reserved for conversations indicating an imminent and credible risk of harm to others. (OpenAI)
That may be a sensible framework.
But it is still an extraordinary power.
And extraordinary power deserves extraordinary transparency.
The real danger is not AI surveillance alone
The greater danger may be opaque AI surveillance.
There is an important distinction between two situations.
In the first, law enforcement seeks access to a user’s conversations through a legal process.
In the second, the company itself identifies a threat and voluntarily contacts law enforcement.
These raise different legal and constitutional questions.
The first concerns the power of the state to obtain private information.
The second concerns the discretion of a private intermediary to decide when private information should enter the criminal-justice system.
That distinction matters.
A society may have detailed rules governing warrants and subpoenas while still giving technology companies significant discretion to decide when they believe an emergency justifies disclosure.
As AI becomes more personal, that discretion will become increasingly important.
The issue is no longer simply:
Can the government read your data?
It is increasingly:
Can the company decide that your conversation is dangerous enough to bring the government into it?
Freedom includes the freedom to have ugly thoughts
There is another danger in treating every disturbing conversation as evidence.
A free society does not protect only pleasant thoughts.
People get angry.
They fantasise.
They exaggerate.
They explore disturbing ideas in fiction.
They experience impulses that frighten them.
Sometimes, talking about those impulses is precisely what prevents them from becoming actions.
A person who tells an AI:
“I’m scared that I might hurt someone.”
may not be announcing a crime.
They may be asking for help.
If people come to believe that admitting a violent thought automatically creates the possibility of a police report, some may stop talking.
That would be an unfortunate result.
The goal of AI safety should not be to make users hide dangerous thoughts.
It should be to help distinguish between expression, distress and preparation.
Those categories must not be collapsed into one.
OpenAI itself has recently said that conversations about violence can be fictional, historical, political, personal or potentially dangerous, and that the challenge is to determine which cases indicate genuine real-world risk. (OpenAI)
That is the right problem to focus on.
Not every violent sentence is a threat.
But not every threat begins with an explicit declaration.
We need rules before this becomes normal
The solution is not to prohibit AI companies from reporting credible threats.
That would be irresponsible.
Nor should companies be given an unlimited mandate to monitor private conversations for anything they consider suspicious.
That would be worse.
What is needed is a narrow and publicly understandable framework.
A serious escalation to law enforcement should normally require several factors:
First: credible evidence of real-world intent. Violent language alone should not be enough.
Second: seriousness of the potential harm. The threshold should focus on death or serious physical injury, not unpopular speech or ordinary emotional expression.
Third: context. A conversation should be assessed as a whole, not by a single alarming sentence.
Fourth: evidence of preparation or immediacy. Specific targets, means, timing or other concrete indicators should carry far more weight than abstract fantasies.
Fifth: meaningful human review. A person should not find themselves reported to police because a classifier generated a high-risk score.
OpenAI’s current public policy reflects several of these principles, including specialised human review and a threshold of imminent and credible harm to others. (OpenAI)
The next question is whether these safeguards should remain purely internal company policy.
They probably should not.
Private rules are not enough
A company can change its safety threshold.
It can alter its detection systems.
It can redefine what counts as credible.
It can decide to disclose more information—or less.
That may be acceptable for ordinary product decisions.
It is less comfortable when the decision may determine whether police begin investigating a person.
There should therefore be greater independent oversight of how AI companies handle emergency disclosures.
Not public disclosure of sensitive conversations.
Not a public database of reported users.
But meaningful accountability.
Companies could publish aggregate information about:
- the number of cases escalated for serious review;
- the number of accounts banned for credible threats;
- the number of emergency referrals to law enforcement;
- the general criteria used to distinguish fantasy from credible planning;
- and the safeguards used to prevent automated false positives.
Transparency will not eliminate mistakes.
But it would allow society to ask whether the system is proportionate.
At present, the public debate is often driven by tragedies that emerge after something goes wrong.
That is a poor way to govern.
A new category of privacy may be necessary
Perhaps the deepest problem is that the law still tends to treat AI conversations as another form of digital data.
That may soon become inadequate.
A search engine knows what you are looking for.
An AI assistant may know why.
It may know your relationship problems, career fears, financial worries and private obsessions. As AI systems develop longer memory and become more deeply integrated into daily life, the amount of intimate context they hold may increase dramatically.
This suggests a need for something stronger than the ordinary idea of data privacy.
Call it conversational privacy.
The principle would not mean absolute secrecy.
If there is compelling evidence that someone is about to kill another person, society may reasonably expect intervention.
But the default should be clear:
Intimate AI conversations deserve a presumption of confidentiality, with narrowly defined exceptions for genuine emergencies.
That would better reflect the reality of how people now use these systems.
The social contract has to be honest
Perhaps the most uncomfortable fact is also the simplest.
People are beginning to treat AI as something close to a confidant.
But an AI system is not a confidant in the traditional sense.
It is software operated by an institution.
That institution may have obligations to users, shareholders, regulators, courts and the public.
And in rare cases, it may decide that keeping a conversation private is more dangerous than disclosing it.
The recent cases involving Zhou and Tumbler Ridge illustrate the two sides of that power.
In one case, an AI company’s intervention reportedly helped bring an alleged threat to the attention of the FBI. (조선일보)
In the other, OpenAI decided not to refer a user to law enforcement before a devastating attack—a decision that later prompted public apologies, lawsuits and demands for accountability. (Reuters)
Neither case gives us an easy answer.
Together, however, they reveal a new reality.
AI companies are becoming something more than software providers.
They are being pushed into the role of gatekeepers between private thought and public danger.
That role may sometimes save lives.
It may also create new forms of surveillance and private power.
The challenge for democratic societies is not to choose between freedom and safety as though one can simply replace the other.
It is to build rules strong enough to protect both.
Because the most important question is no longer whether an AI can understand what we say.
It is this:
When the machine believes it understands what we might do next, who should have the power to act?
